M P Ra Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. Prediction markets can be run continuously, and they motivate participation and participants to reveal their true beliefs. On the other hand, they need many participants and clear outcomes in order to determine pay-offs. Moreover, translating knowledge into a price is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists’ knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts but may also lead to conformity due to group pressure. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the “black box” nature of prediction markets. We consider that, half a century after its original development, Delphi is under-utilized. Much can be done to improve upon traditional group meetings. As Armstrong (2006) showed, it is difficult to think of a structured approach that would not improve on the predictions and decisions made in traditional meetings. Rowe (2007) presents evidence that, in comparison with traditional meetings, the Delphi technique can improve forecasting and decision making. How does it do that? If conducted properly, Delphi greatly improves the chances of obtaining unbiased estimates and forecasts that take full account of the knowledge and judgment of experts. A third method for aggregating individual judgments, prediction markets, is based on the idea that markets provide forecasts by giving people an opportunity to benefit financially or otherwise by buying, or selling, when the current price seems too low, or high, given the knowledge they possess. The market price thus reveals the aggregate knowledge of market participants. We discuss the relative merits of Delphi and prediction markets. How Delphi Has Been Used The Delphi procedure has been around since the 1950s. To assess its use, we conducted a Google search for “Delphi AND (predict OR forecast)”. This yielded 805 unique sites out of a total of 1.4 million, showing that some people have paid attention. Using the same keywords, we conducted searches in the Social Sciences Citation Index and the Science Citation Index Expanded to assess what has been happening to researcher interest in Delphi over the years. We identified altogether 65 relevant items; 1 from the 1960s, 8 from the 1970s, 3 from the 1980s, 21 from the 1990s, and 32 so far this decade. When we searched for “Delphi forecast of” and “Delphi forecasts of”, we found 42 unique applications of the Delphi technique. Most of them (43%) were business applications. These included forecasts for: • the Argentine power sector • broadband connections • dry bulk shipping • leisure pursuits in Singapore • rubber processing • Irish specialty foods, and • oil prices. Forecasts of technology were also popular (36%), these included forecasts about intelligent vehiclehighway systems, industrial robots, intelligent internet, and technology in education. Finally, 21% of applications were concerned with broader social issues such as the “urban future” of Nanaimo in British Columbia, and the future of law enforcement. We also found about 1,000 unique items using a Google Scholar search for the “Delphi method” and either “forecast OR predict OR estimate.” This also suggests that Delphi is widely used. We have ourselves employed Delphi for problems ranging from forecasting prisoner number, to choosing between regional development options, to predicting outcomes of political elections, to deciding which applicants should be hired for academic positions, to predicting how many meals to order at conference luncheons.